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Forewords 


A foreword from GREG LIPPMAN, 
Founder of the National Campaign for Highest-Need Students 


ow do we know when one school is better and more effective than others? And 

perhaps more crucially, given the challenges and crises of the current moment, what 

makes a school focused on the highest-need and most vulnerable students in our 
communities better than the rest? 

In 2018, I founded the National Campaign for Highest-Need Students, an effort to ensure 
that schools that are having the most impact on student achievement for our least successful 
student populations are identified and recognized for their efforts, and that some of the most 
innovative, impactful educators in the country are supported in their work to spread their 
models to as many students and systems as possible. 

But before any of that can happen at scale, we need to address a crucial question the answer to 
which has so far proven elusive: which schools work best for highest-need students? And why? 

In our current accountability systems, even schools that are doing the best work locally (or 
even nationally) with struggling students are nearly always labeled “failing” schools. And the 
more high-need students these schools serve, the more their overall performance metrics suffer. 
In order to disrupt these systems and eliminate these blind spots, a better way of comparing 
schools is necessary, one that ensures that schools are compared with schools that are serving 
student populations that are demonstrably similar, and that are showing bend-the-curve 
impact with those students. 

That is what the School Needs Index 1.0, the subject of this working paper, is designed to 
do. The School Needs Index uses previous student academic performance, as well as a host of 
demographic and economic indicators, to yield a nuanced profile of the level of student need 


at a school. And while there is much work still to be done here, this index is a crucial first step 
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toward giving system-level decision-makers the tools to accurately assess a school’s true impact 
on its neediest students. 

I want to thank the Walton Family Foundation and the Bill & Melinda Gates Foundation for 
their early support of the School Needs Index project, as well as the Scott Family Foundation, the 
Charles and Helen Schwab Foundation, Karen and Michael Gordon, the Margulf Foundation, 
the House Family Foundation, the Sobrato Family Foundation, and the City Fund for their 
support of the broader work of the National Campaign for Highest-Need Students. 

I also especially want to thank and recognize all of the educators, advocates, and funders 
who have contributed to its development, including the members of this report’s Advisory 
Committee listed above. It is the schools and models that have resulted from their tireless years 
of effort that the School Needs Index is designed to identify, and it is their work that is our best 
hope for a public education system that provides highest-need students and their families the 
schools and opportunities that they deserve. 


A foreword from Education Analytics & the CORE Data Collaborative 


he CORE Data Collaborative and Education Analytics partnered with the National 

Campaign for Highest-Need Students as part of our common focus on an equity and 

data driven understanding of public schools. A “one-size fits all approach” to schools 
misses one of the most important parts of engaging in the continuous improvement of the 
education system—understanding and addressing variation in student need in our schools. 
At the same time, an approach that considers every school to be totally unique would make 
it almost impossible to engage in the strategic and thoughtful support and management of 
schools, and every conversation would potentially devolve into unique perspectives on student 
need. 

We believe the School Needs Index described here is designed to split the difference between 
these two extremes. By anchoring need in an outcome of interest like students’ ability to read, 
write and comprehend, a needs index can help us look across schools to understand the varia- 
tion of student need in a common, digestible and actionable way. We look forward to moving 
from the exploratory work described in this paper toward proof of concept efforts in areas like 
the city of Oakland. 
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Identifying Schools Achieving Great Results 
with Highest-Need Students 


Catalyzing Action to Meet the Needs of All Students 


INTRODUCTION 


ven before the coronavirus pandemic, educators across the country reported students 
arriving on the first day of class needing unprecedented levels of support to thrive. Our 
education system applies labels to describe many of those needs—homeless, pregnant, 
students with disabilities, eligible for free and reduced-price lunch, and so on. But those labels 
capture just one small part of who students are, and speak nothing of what they can accomplish. 

At the same time, too few schools today stand ready to help all students meet their promise. 
In fact, crude accountability systems often provide disincentives for the schools that want to 
serve students with the greatest academic needs. Meanwhile, schools that serve such students 
exceptionally well can go unnoticed—or even be labeled as “failing.” As a result, other schools 
have no way of learning from and replicating their achievements. In fact, schools rarely even 
know what other schools serve a population like theirs. Schools serving the highest-need stu- 
dents—and their successes with such students—are too often invisible. 

This was the experience of ACE Charter Schools, a network of schools serving highest-need 
students in San Jose, California. In response, ACE’s founder, Greg Lippman, held a national con- 
vening in 2018, pulling together educators serving highest-need students along with funders 
and advocates from across the country. The convening sparked the National Campaign for 
Highest-Need Students, an initiative to ensure that highest-need students have schools that 
work for them. 

To address the invisibility of schools serving highest-need students well, ACE partnered with 
Education Analytics, the CORE Data Collaborative, and Public Impact to change the definition 
of school quality, making visible the success that schools are achieving with these students. 
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This report represents our first steps toward that goal: 


mu The “School Needs Index.” Section 1 explains our methodology for measuring the extent 
of support that students need to thrive academically. By calculating a School Needs Index 
for each school, we can help educators, policymakers, and the public gain a clearer un- 
derstanding of the level and type of support that each school’s students need to succeed. 

m Success Measures. Section 2 explains methods to apply the School Needs Index to identify 
schools achieving outstanding results with highest-need students. With this kind of flashlight, 


we can point the way for many more schools to serve highest-need students exceptionally well. 


The report also includes a series of appendices providing additional information about our 
analysis and the technical details that will allow other researchers to replicate our methodology. 

Our hope is that the methods presented here will be immediately useful to educators and 
the agencies that oversee schools, providing tools to improve understanding of schools’ chal- 
lenges and their successes. At the same time, we acknowledge—and discuss extensively in 
this report—that the methods described here are “1.0.” Our intent is to improve on them over 
time, and to invite others to join a national data collaborative with that aim. As we increasingly 
identify schools achieving success with highest-need students, we can turn to the critical work 
of understanding their success and sharing their practices widely, vastly increasing the share 
of highest-need students who attend schools that work for them. 

Then we could apply what we learn to better serve highest-need students at scale. 


The School Needs Index 


When we talk about the level of student need at a school, we typically fall back on some familiar 
metrics. What percentage of students are eligible for free or reduced-price lunch (a conventional 
measure of poverty)? What percentage are English language learners? What percentage receive 
special education? 

But these metrics mask as much as they reveal. Consider two schools, both with 100 percent 
of their students eligible for free lunch. What if you opened the schoolhouse doors and learned 
that the first school enrolled exclusively students who were involved in the juvenile justice 


system, half of them living away from their parents? What if the second school’s students came 
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from households—most of them with two parents—in the low-income but relatively stable 
neighborhood surrounding the school? 

You'd quickly conclude that when it comes to student need, defined as the level of support a 
school must provide its students to be successful, these two schools are actually very different. 
The 100 percent free lunch measure wasn’t meaningless—both schools faced challenges you 
wouldn't see across town in a wealthy neighborhood school—but you'd want a way to acknowl- 
edge the differences in the need for support in each school. 

Understanding that schools serve students requiring different levels of support is just a start. 
We can use that understanding to identify schools that are similar to one another, with an eye 
toward spotting schools achieving exceptional results with highest-need students. And we can 
use these measures of success to set ambitious but realistic targets for schools to shoot for with 


their highest-need students. 


Calculating a School Needs Index 


To understand the level of support needed by each school’s students, we created a “School 
Needs Index.” Our data set included dozens of student characteristics at the individual level. 


We categorized those into four “domains” (see the appendices for a full list of variables): 


1. Student engagement (e.g., chronic absenteeism, suspension) 
2. Demographics (e.g., migrant, new to the school, specific disability status) 
3. Academics (e.g., prior ELA/math test scores, grade-point average) 


4. Economics (e.g., homeless, estimate of family income) 


We wanted to pull all of this information into a single score for each student, indicating the 
student’s likely need for support. To do that, we needed to know how important each charac- 
teristic was in predicting students’ success in school. If a characteristic was very predictive, we 
wanted to give it a great deal of “weight” in our scoring system. If it was less important, we 
wanted to give it less weight. As described in the appendices, we used statistical techniques to 
figure out these weights, based on the experience of the students in our data set and consider- 
ing a variety of school-level factors. 

One challenge was deciding what measures of “student success” to use in our model. For 
this analysis, we focused on success in math and English language arts (ELA) as measured by 
test scores. To be clear, we did not choose this path because of a belief that scores on math and 
ELA tests are the ultimate measure of student and school success. The extent to which students 
thrive throughout their lives, including in life beyond school, is our ultimate interest. And other 
student outcomes while in school, such as the development of personal competencies and social 
and emotional health, are also important. We chose to use math and ELA performance as our 
outcome metrics because they are leading indicators of how well students thrive after they 
leave school, and because they are readily available in the CORE data, as well as comparable 


data sets nationwide. 
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As we expand this work, we will expand the range of outcomes we analyze. And as we do, 
the specific weights attached to each student characteristic will likely shift as well. What we 
want to focus on here is our methodology for determining weights and the resulting index, 
rather than the specific weights we derived in this round of analysis. 

The method we developed allows us to assign each student a score from 0 to 100, with a higher 
number representing a higher level of student support needed. To calculate a school-level School 
Needs Index score, we then averaged individual student support scores across the school. 


Visualizing School Needs Index Data to Answer Key Questions 


After computing the School Needs Index score for 3,000 schools in our data set, we developed 


visualizations to help us address four key questions: 


1. What is the level of student need (School Needs Index score) for any particular school? 

2. How do student needs vary across schools? 

3. How do student needs vary by domain? 

4. How do student needs vary among schools that seem similar on the surface (e.g., similar 


proportion of students eligible for free and reduced-price lunch)? 


Although the next few pages linger over the Success Index, it is important to reiterate that 
the Success Index is not an end unto itself. Rather, the index provides a tool through which we 
are able to measure a school’s success in a way that acknowledges differences in student needs. 
That allows us to identify and learn from comparable schools that are achieving exceptional 
results for students relative to other schools serving students with similar needs. The rest of 
this section focuses on answering the four questions above as a way to consider which schools 


are truly “comparable.” 


What is the level of student need (School Needs Index Score) for any particular school? 
At its most basic level, the School Needs Index score described in this report aims to quantify 
the level of academic support that students need in their school. Figure 1 displays that need 
as a stacked bar graph, showing both the overall level of need (the top of the bar) and the 
relative need of each of the four domains—academic, demographic, economic, and student 
engagement—for one school. 

In the example in Figure 1, Schools A’s School Needs Index score is 87 ona scale from 0 to 100. 
The academic domain accounts for 74 of the 87 points, followed by demographic and student 
engagement (5 points each), and economic factors (3 points). The academic domain—students’ 
prior academic performance—dominates this analysis because the outcomes we used in de- 
veloping the index were academic outcomes, specifically ELA and math scores. As we use a 
broader set of outcomes, we expect the relative importance of domains (and specific character- 


istics) will change. 
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Figure 1. School Needs Index Score for School A 
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Figure 2. School Needs Index Score for all Schools in School A's District 
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How do student needs vary across schools? 


Figure 2 allows us to put each school’s School Needs Index score into context. In Figure 2, each 
bar represents a single school within California’s CORE data set. A similar figure could be 
created for any group of schools—every school in a particular district, every charter school in 
a state, and so on. It shows that the level of need for academic support varies widely within this 
group of schools, with some schools having a School Needs Index score three times as large as 
those on the other side of the chart. The red line shows the average level of need, allowing us 


to see that many schools fall above and below this midpoint. 


How do student needs vary by domain? 


Although the academic domain accounts for much of a school’s School Needs Index score, it 
is certainly not the only domain that matters. To focus on the other domains, we re-ran the 
analysis, removing the academic domain from the model. See Figure 3 for the result, which 
highlights the variation in the other domains, with demographics being the most predictive 
domain, followed by economics and student engagement. Figure 3 also shows, however, that 
the composition of student support needed differs from one school to the next. 

As a result, schools with a similar overall School Needs Index score may vary in the types 
and levels of student needs they serve. The three schools in Figure 4, for example, have an 
almost identical overall score (between 82 and 84), but differ in the three domains that make 
up the measure. School F and School G have about 65 points in the demographic domain and 
only 7 and 10 points on student engagement, respectively; in contrast, School H has 52 points of 
demographic domain and 18 points of student engagement. To better illustrate the differences, 
we zoom in on three schools with a similar level of overall need. This image shows how the 


composition of need can vary significantly by school 


How do student needs vary among schools that seem similar on the surface? 


In contrast with Figure 4, where the schools had a similar overall need, Figure 5 (page 12) 
includes four schools that look very similar on a particular measure of need: 100 percent of 
students at all four schools receive free or reduced-price lunch (FRL). Despite this surface sim- 
ilarity, these schools are dissimilar with respect to overall need. While Schools L and N have a 
need of more than 85, Schools K and M score only 50 and 59, respectively, with the difference 
stemming from the demographic domain. 

Each of these visualizations answers a different question, illustrating the potential power of 
the data to help educators, and the organizations that support them, understand the type and 
level of support needed to serve their students well. With that information in hand, it becomes 
possible to identify comparable schools, making “apples-to-apples” comparisons possible. As 
discussed in Section 2 (page 15), we can then identify schools achieving extraordinary success 


compared with comparable schools—and learn from them. 
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Figure 3. School Needs Index Score for all Schools in School A's District: Three Domains of Need 


100 


Domains and weights 


m Student Engagement 


| | 
50 i | i 
‘i m Economic 


m Demographic 


School Needs Index Score 


Schools 


Figure 4. Schools with similar School Needs Index scores but different types and levels of need 
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Figure 5. Schools with 100 percent FRL students but different types and levels of need 
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Takeaways from the CORE data: School Needs Index 


The primary purpose of this report is to shed light on the methodology used to create the 
School Needs Index and the related success measures (described in the next section, beginning 
on page 15), rather than to describe specific findings from our analysis of the CORE data. Still, 
we want to note two important takeaways from our analysis with the caveat that results could 
differ significantly in different geographies, with additional years of data, or with different 


student outcome measures. 


Consistency in predictive factors 

First, the factors that predict ELA and math scores in the CORE data are largely consistent 
across grade levels and subjects. While predictive factors might differ using other outcomes 
that we hope to examine in the future (such as measures of post-secondary success), the con- 
sistency with which our model identified the same support factors across models gives us 
confidence that the School Needs Index is widely applicable across grades and subjects. 


Prior performance—the most predictive factor 


Second, students’ prior test scores were by far the strongest signals of support needed when we 
analyzed the impact each domain seemed to make on student success as defined by achieve- 
ment in ELA and math. This was not surprising; prior test scores capture the accumulated 
needs of students’ lives, and those observed and unobserved factors would not change dra- 


matically from year to year. 
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The fact that the “academic” domain is the largest factor in the School Needs Index does 
not mean, however, that prior performance dictates a student's (or school’s) academic destiny. 
On the contrary, we find that schools (and thus students) with similar academic performance 
in the past can go on to achieve very different academic results, with some greatly exceeding 
our model's predictions. Nonetheless, knowing a student's prior academic performance is im- 
portant because it signals the level of support that that student needs. And it is important to 
know the prior academic performance of the students entering a school for the same reason. 
Today, most methods used to report school characteristics fail to provide information about 
the academic records of a school’s incoming students. This research shows that this is a major 


missing piece of current systems that needs addressing. 


Data Considerations: School Needs Index Score 


Our analysis highlighted several considerations for other researchers applying our methodol- 


ogy to create a School Needs Index. 


Isolating the effect of the school on student need 

Ideally, a school’s School Needs Index score would entirely reflect factors outside of a school’s 
control, such as family income and students’ experiences in prior schools. We were not able to 
reach that ideal in our analysis, however, because some of the indicators that go into estimating 
a school’s need in a given year reflect what happened to students in the same school the prior 
year. One could imagine a scenario in which a school made substantial progress on math and 
reading achievement with students from one year to the next, sending the school’s School 
Needs Index score down, while another school that started with identical students but did not 
make similar academic progress would have a higher School Needs Index score. 


Weighing the cost-benefit of additional variables 

Some education agencies may want to include additional indicators that require significant 
resources to collect, such as information related to adverse childhood experiences. In those 
instances, they will need to weigh the benefit of those data points (what they tell us) with the 
time and expense of collecting them, as well as any student privacy concerns that arise. Further 


studies could shed light on the cost-benefit tradeoff of specific metrics. 


Creating new variables 

In addition to (or in lieu of) collecting additional variables of student need, policymakers and 
practitioners could create new variables by tracking changes in a particular variable over time. 
For example, they could create an indicator of the consistency of a student’s FRL status, bottom 


decile performance, or student transiency and mobility over a particular time period. 


Including special education data appropriately in the model 


Students in special education are highly diverse and vary widely in their educational needs. 


As a result, simplistic metrics like the percentage of a school’s students with individualized 
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education programs (IEPs) are not an adequate measure of need at the school level. Here, we 
explore some possible approaches to quantifying disability-related need. Though all have lim- 
itations, future research can shed light on the best methods for using these metrics. 


= Disability codes. The U.S. Department of Education (USDOE) defines 13 disabilities for 
which students can receive special education services (such as specific learning disability, 
emotional disturbance, and traumatic brain injury),! and requires states to report annually 
on the number of students receiving services in each category. Some states even include 
additional categories. For example, in the analysis reported here, we were able to cate- 
gorize students based on the CORE data’s disability groups (see Appendix, page 20). In 
theory, some of these categories could be deemed more “high-need” than others, with the 
percentage of students in such categories serving as a better measure of need. In practice, 
however, student needs are also diverse even within these narrower categories of student 
disabilities. 

a Educational environment. USDOE requires states to report the percentage of students 
receiving services in general education (inclusion) versus separate environments. Since 
students with more severe disabilities often spend more time in specialized settings, 
these data could potentially serve as a proxy for a school’s level of disability-related need. 
Inclusion rates are not just a function of need, however. They also reflect policy choices at 
the state, district, and school levels. Two schools with identical student populations could 
have very different inclusion rates because of their divergent approaches to educating 


students with disabilities. 


This phenomenon plays out clearly when comparing inclusion rates at the state level—al- 
most 84 percent of special education students in Alabama learned in a general education 
classroom for 80 percent of the day or more in the 2017-18 school year, while in New 
Mexico, Montana, New Jersey, and Hawaii, the same was true for less than 50 percent of 
all such students.” There is no reason to think these differences reflect a vastly lower level 


of need in Alabama; policy and resources are likely the driving forces. 


= Funding amount. States establish and publish funding formulas based on disability cat- 
egories. Funding levels are based on estimated costs, placement guidelines, and other 
administrative factors. Similar to inclusion rates, these formulas vary widely based on 


policy choices, and therefore are not necessarily good indicators of school-level need. 


a IEP data. Individualized education programs, required for all students receiving services, 
could potentially provide standardized information about special education students 
across states. All IEPs include details about the related services that students receive (such 
as physical therapy or speech-language pathology services), whether they have curricular 
accommodations or modifications (such as, has their learning goal been changed?), or if they 
are on a diploma or alternate certificate track. All schools may not compile this level of data 
in an easily accessible database, however, even though each IEP is required to contain the 
information. If these or other variables could be collected across schools, indices may find 


that they better demonstrate a consistent level of need for special education students. 
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Some combination of disability codes, education environment data, funding categories, 
and IEP data could yield a more accurate measure of academic support needed than any 
one source alone, but in light of these challenges, more research and testing is needed. 


Methodologies for Measuring School Success and 
Identifying Bright Spots 


In the previous section, we presented a methodology for defining the academic sup- 
port a student or school needs (a School Needs Index) based on four kinds of student 
factors—academics, demographics, economics, and student engagement—for the 
purpose of identifying “comparable” schools. In this section, we turn to measuring 
school success—how well schools improve the outcomes of their highest-need stu- 
dents relative to those comparable schools. 

We created this methodology with the goal of uncovering “bright spots,” schools 
that seem to achieve higher levels of student growth than our model would predict 
given the needs of their students. Our hope is that by identifying these bright spots, 
the field can also identify the core practices that set them apart, then replicate those 
practices in a more diverse group of school settings serving highest-need students 


across the country. 


Measuring School Success 


To calculate school success, we created what’s known as a “value-added model.” The 
model predicts the ELA or math performance for a school’s students based on how 
schools serving students with similar needs tend to perform from one year to the next 
(though we aim to include multiyear longitudinal data in the future). The model then 
compares actual student growth at the school to the predicted growth. The more that 
actual student growth at a school exceeds predicted student growth, the higher its 
score and the more “successful” the school (or, the more “value” the school “adds”). 
Moreover, the model allows observers to make an apples-to-apples comparison of stu- 
dent achievement in schools with similar levels of student need. In effect, this levels 
the playing field between schools, giving a clearer view of their performance con- 


sidering their different needs. See Appendix 1 for more detail on our methodology. 


Visualizing School Success Data to Answer Key Questions 


As we did with the School Needs Index, we again developed visualizations to help 
shed light on key questions: 


1. Which schools serving significant proportions of highest-need students seem 
to be most successful? 
2. Which schools seem to be most successful serving a particular group of 


high-need students? 
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Which schools serving significant proportions of highest-need students seem to be most 
successful? 
Figure 6 plots each school as a single dot. The farther to the right a school’s dot is, the more sup- 
port its students need based on the School Needs Index. The higher a school’s dot is, the greater 
the growth of its students—in this example, in fourth-grade ELA. So, a school with a dot in 
the upper right of the chart is both relatively high-need and high-growth in fourth-grade ELA. 
It is clear from this chart that schools at all levels of need have a wide range of success in 
fourth-grade ELA growth. For example, schools A, B, and C all need high levels of support. But 
school A’s fourth-grade ELA growth is very high, while B’s is more average, and C’s falls below 
that of most schools. In fact, School A’s School Needs Index score is among the highest in the 
data set, but it has one of the highest levels of growth. We consider schools like School A “bright 


spots” that may be able to share promising practices and approaches driving their success. 


Which schools seem to be most successful serving a particular group of high-need students? 
As noted earlier, our model allows us to measure school success with a particular school group 
or student need. The result is Figure 7, which shows the value-add a school has on different 
student groups within the school. 

Figure 7 depicts, for one school, three levels of detail for math that increase from left to right. 
The “Overall” bar chart on the left shows the school’s average success in raising test scores for 
students overall, scaled as a percentile. This school’s success in its impact on all students is 
approximately at the 87th percentile of all schools—that is, its impact is as high as or greater 
than that of 87 percent of all schools in the data set. 

The “By Need Level” graphic in the middle column shows the next level of detail. We divided 
the school’s students into three groups based on their individual support scores. The graphic 
shows that this school was highly successful with its highest-need students, and moderately 
successful with its middle- and lowest-need students. 

The “thought bubble” in the right-hand column shows the school’s success with students in 
the highest-need group with each domain of need. For example, this school does especially 
well with students with high academic and student engagement needs (those in the top third 
of all students in these domains). It does moderately well with students with high economic 
and demographic needs. Though not shown here, the analysis could drill down even further, 


showing the school’s success with more specific subgroups within domains. 


Takeaways from the CORE data: School Success 


As noted earlier, this report does not aim to dwell on the results of our analysis of the CORE 
data. But two takeaways seem worth highlighting with the caveat that they may not hold true 


using other or additional years of data. 


School success scores are similar regardless of need 
We explored how student success varied by dividing schools into five equal-size groups (“quin- 


tiles”) based on their School Needs Index score and finding the median Success Score for each. 
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Figure 6. School success by level of need 
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Figure 7. School success by three levels of detail 
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Through that process, we found little difference between quintiles (see Appendix 2, page 22). 
These findings are heartening; they suggest that on average, schools enrolling many students 
who require high levels of academic support achieve the same level of success as schools that 
enroll students who require less support. They also validate the role of a School Needs Index 
score like the one we created to separate the impact the school is having from the students the 
school is serving. In contrast, traditional accountability systems generally compare schools 
based on student achievement, irrespective of differences in student need. Yet schools serving 
many high-need students who arrive performing far below grade level tend to have lower 
proficiency rates: Even if they are producing strong student growth, that growth may not be 


sufficient for enough students to reach proficiency. 


More outliers at higher School Needs Index scores 

We also observed that among schools with higher School Needs Index scores (indicating higher 
levels of student need), we see more outliers in both the positive and negative directions. This 
suggests that the school in which the highest-need students enroll has a much bigger impact on 
their performance outcomes than for their peers with the lowest needs—all the more reason to 


identify the most successful among them and the practices underlying their success. 


Data Considerations: School Success 


Our analysis highlighted several considerations for other researchers applying our methodol- 


ogy to measure school success. 


Identifying outliers 

One primary goal of our work is to identify schools with comparable levels of student need, and 
which among them were the bright spots—schools with significantly greater success than their 
peers. To identify such outliers, however, it is critical to apply consistent decision rules, including: 

a A threshold for success. This is the level of success needed to qualify as a bright spot, 
such as being at or above 70th percentile of growth for comparable schools, or showing 
statistically significant higher growth than the mean for comparable schools, etc. 

a The number of years at or above the threshold. Meeting the threshold for success one year 
may be due to atypical circumstances—so then how often must a school meet the threshold 
to be a bright spot? And must those years be consecutive, or could it be something like two 
out of the last three? 

= The number of outcome metrics at or above the threshold. An analysis might look for 
outliers on a specific success metric (such as ELA performance or post-secondary readi- 


ness), or across multiple success metrics. 


As an example, an analysis could define bright spots as schools at or above the 95th percentile 
(relative to schools with a comparable School Needs Index) in math and ELA for three consec- 
utive years. To reach the best option, analysts should experiment with different combinations 


to yield a reasonable number of schools. 
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We note that these decision rules constitute only the first stage of the process; the real value 
is in understanding how and why some schools shine. Educators and policymakers must un- 
derstand what bright spots do differently to enable them to replicate the results and benefit 


more students. 


Limits of “value added” 


We use a standard value-added methodology to estimate school success, but the limitations that 
apply to all value-added models apply here (for example, the need for multiple years of data to 
get a strong signal of the school’s success). Notwithstanding those limitations, the value-added 
methodology here provides a much stronger signal of school success with highest-need stu- 
dents than many commonly used measures. Future work can improve on the analysis here by 


expanding the definition of success beyond ELA and math performance. 


Sample size 

Analyzing school success for specific subgroups of highest-need students requires large sample 
sizes. Many data sets will be much smaller than CORE’s, pointing to the importance of forg- 
ing data collaboratives through which research can analyze multiple data sets with similar 


methods. 


Next Phases of the Search 


The School Needs Index and School Success metrics presented here mark only the first stage of 
a larger effort to improve how we serve the highest-need students. It is our sincere hope that 
this work will help independent researchers and education agencies identify schools serving 
groups of highest-need students exceptionally well so that we can learn and scale up their 


approaches. In future work, we will (and encourage others to): 


a Explore additional outcome metrics beyond math and ELA outcomes, including the many 
other academic and nonacademic metrics that so many schools serving high percentages 
of highest-need students already use 

u Use more longitudinal data to find indicators of need that appear only over the course of 
years, and that can be based solely on factors outside the influence of a student’s current 
school 

a Add additional indicators of student support needed by more effectively capturing con- 
cepts like the severity of disability or the depths of a student’s poverty or trauma 

u Apply the analysis to different data sets to see what might be generalizable beyond CORE 


We conclude this phase on an optimistic note. Some of the students in the data faced so 
many challenges, and though they needed tremendous amounts of support, it was not more 
than schools could give. For every group of students, a group of schools successfully provided 
the support they needed. The challenge moving forward is to learn from these schools and 
spread their practices so that many more of the highest-need students can excel. 
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Appendix 1: Methodology 


Creating a School Needs Index 


The model can be written as: 

ba € + er B; a C iste 
where: 
La Mae 
iin subject s in school k in year t, 


is the outcome test score in math or ELA for a student in the outcome year, for student 


w €is the intercept of the model, 
m X,,,isa vector of characteristics for student i in year t, 


m ¢,,,is a student-level error term. 


We estimated this regression using an errors-in-variables (EIV) method that accounts for 
measurement error in any student characteristics that may need correction, such as prior 
test scores.? We used conditional standard errors of measurement for the Smarter Balanced 
Assessment Consortium (SBAC) math and ELA assessments. 

After estimating the model, we used the coefficients B to create a prediction of the outcome 
for any individual student. To do this, we multiplied the coefficients we estimated for student 
characteristics by the observed values of the student's characteristics. 

Yiu = E+ Xin B, 

This prediction is our best estimate of the student’s math or ELA test score at the end of the 
school year, absent the impact of an intervention (being in a high-growth school or receiving 
additional support). The lower a student’s predicted score is, the higher a student’s need for 
academic support. Correspondingly, we define school need as the average predicted score of 
the school’s students. 

We categorized the student characteristic variables X,,, into domains to investigate how much 
different kinds of variables contribute to overall need. To do this, we broke X,,, into “domains,” 
which we will call X',, and X*,, (there can be any number of domains). As an example, in 
this analysis we partitioned variables into four domains: student engagement, demographics, 
academic, and economic. Then we created a “partial” prediction of the outcome for a student 
using just that subset of variables. The following formula shows the domain-specific partial 


prediction for domain 1 and domain 2. 


iskt ikt Fs 
For domain 1, the formula allows us to calculate the proportion of differences in the outcomes 
that is explained by the variables in domain 1 (similar to an R-squared measure for domain 1). 


We can think of this as the “effective weight” of domain 1’s contribution to the overall prediction. 


sD) 


R200 


SPY, SD 
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To put the predictions on a more intuitive scale, we re-scaled the sum of the components into a 
“Support Index” in which values fall in a range between 0 and 100 and tend to be concentrated 


mostly between 30 and 70. 


NIP! = ——_—._ + 50 * R20 


The total Support Index score at the student level is the sum of the domain components. 
NI= NI?'+ N. - 


In order to investigate the school-level total Support Index, we simply averaged the individual 


support indices of all students in that school for that subject. 


Measuring school success 


Our model to measure school success controlling for student needs can be represented as shown. 


skt Yoset™ (é + Xie B,) 


or 


n~ 


Oe, = Y: 


skt iskt — Ysts 


School success with students with specific needs 


We adjusted the model to determine school success in supporting specific student needs as 
well. In the model, we represent the subgroup of interest as subgroup g. Then we take the 
mean difference between the end-of-year test scores and measure of need by school, subject, 
and grade while only including students classified in group g. The school effect of the mean 
difference between test scores and measure of need for the group by school is the measure of 
how well that school does with students in this group. This model can be represented as shown. 


Q srot = Le 7 re + Xirgt B,) 


or 


n 


¥, 
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Appendix 2: Exploring variation in school success by level of 
need 


Figure 1A below shows how the spread of value-added estimates in the school success model 
varies based on the level of student need at the school. It includes data from 1,366 schools divided 
into five quintiles on the X-axis based on student need. On the y-axis, we plot the value-added 
estimate for a given grade and subject. Each group of schools has its own boxplot. These boxplots 
can be interpreted in the following way: 


u The tick line in the middle shows the median value-added estimate for that group 


= The lower and upper lines of the box show schools between the 25th and 75th percentile of 
value-added, respectively. In other words, one quarter of the group is below the box and one 
quarter is above the box, while 50 percent is within the box. 

u The vertical lines that go from the box to the top and to the bottom stop at the 95th and 5th 


percentile respectively. The points below and above those lines are outliers. 


The figure shows that the median value-added estimate (Success Score) for schools was approx- 
imately the same regardless of student need, but that the spread tended to increase as student 


need increased, as evidence in the number and distance of outlier dots moving from left to right. 


Figure 1A. Distribution of school success by need quintile 
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Appendix 3: Variables 


Domain WETatle) (2 1BY-erul o} atoyal 
Math Pretest The student’s math score in the previous year 
Academic ELA Pretest The student’s ELA score in the previous year 
GPA (grade-point average) | The student’s previous year GPA (only for high school) 
Free & Reduced-Price Indicates a student’s free or reduced-price lunch 
Lunch eligibility status 
eee Low Parental Education neue sien primary guardian does not have a 
high school diploma 
Homeless Indicates student has a homeless status 
Foster Care Indicates student has a foster status 
Hard of Hearing/Deafness | Indicates student is hard of hearing or deaf 
Speech or Language Indicates student has a speech or language impairment 
Impairment 
Visual Impairment Indicates student has a visual impairment 
Orthopedic Impairment Indicates student has an orthopedic impairment 
Other Health Impairment _ | Indicates student has other health impairment 
Intellectual Disability Indicates student has an intellectual disability 
Traumatic Brain Injury Indicates student has a traumatic brain injury 
Specific Learning Disability | Indicates student has a specific learning disability 
Emotional Disturbance Indicates student has emotional disturbance disorder 
Autism Spectrum Disorder | Indicates student has autism spectrum disorder 
Demographic (ASD) 


Migrant 


Indicates student participates in the Migrant Education 


Program 


English Language Learner 
(ELL) 


Broken in four categories based on proficiency. ELL 
category 1 is the least proficient. Category 2 is in the 
middle, and Category 3 is the most proficient. Category 
99 is for students who are ELLs, but their proficiency 


level is not known 


New to School 


Indicates student is new to school in current year 


Race Includes African American, American Indian/Alaskan 
Native, Asian, Filipino, Hispanic, Pacific Islander, and 
White 

Gender Includes Female and Male 


Student 


Engagement 


Chronic Absenteeism 


Indicates student was chronically absent in the previ- 
ous year. Chronically absent is defined as when a stu- 
dent has an attendance rate less than or equal to 90% 


School Safety 


Percent of students who feel safe in school based on 


climate surveys (only school average) 


Non-Continuous 


Enrollment 


Indicates student was not continuously enrolled at one 


school in the previous school year 


Suspension 


Indicates student was suspended in the previous year 


In-School Suspension (ISS) 


Indicates student had an in-school suspension in the 


previous year 


Deaf-Blindness and Established Medical Disability variables were available but were excluded because 


too few students in the data set had these disabilities. 


Additionally, we include the following variable taken from the census data: 


Description 


Separate indicators for whether a student’s zip code has 
a mean income in the bottom third or in the middle third 


of all students. 


| DXoyent-ben) Variable 
Economic Mean Neighborhood 
Income 
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